A randomization-based perspective of analysis of variance: a test statistic robust to treatment effect heterogeneity

Fisher randomization tests for Neyman’s null hypothesis of no average treatment effects are considered in a finite population setting associated with completely randomized experiments with more than two treatments. The consequences of using the F statistic to conduct such a test are examined both theoretically and computationally, and it is argued that under treatment effect heterogeneity, use of the F statistic can severely inflate the type I error of the Fisher randomization test. An alternative test statistic is proposed, its asymptotic distributions under Fisher’s and Neyman’s null are derived, and its advantages demonstrated.